Balancing Memory Load in D-CAPE Using Replication-Style Distributed Partitioning of Operator States

Distributed continuous stream query systems have been shown to meet the needs of a range of high-volume applications. However, in some applications the stream queries for these applications may require a wide range of sizes of allocated memory. In such cases it may not be feasible to continue to store the data for any of the largest operators on a single machine. In addition, for the benefit of the users of such a system, we would like to offer as much flexibility in query processing as possible. This includes making full use of resources on the network, even with a highly fluctuating volume and distribution of data. This thesis proposal discusses an unexplored method of partitioning a query operator across several machines, without predetermining a destination for each new input for the operator. The proposed extension consists of the ability of the D-CAPE distributed continuous stream query system to activate a query plan operator on several machines simultaneously, and replicate new input data to each of these operators. Experiments will be conducted to observe the limits of high-volume query processing in D-CAPE with and without these new extensions, and to experimentally optimize these limits through the manipulation of several internal options.